Skip to content

OTA-1578: Disable ResourceReconciliationIssues condition #1216

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged

Conversation

petr-muller
Copy link
Member

Stop maintaining RRI condition and remove it if previously present.

Previously we enabled this experimental behavior for the UpgradeStatus cluster-wide feature gate. We want to use the UpgradeStatus feature as a "vehicle" to promote the oc adm upgrade status CLI but we do not want to promote the RRI with it.

I do not remove the rest of the reconciliation_issues.go code that creates the content we were feeding into the condition because I have ideas about how to make that functionality useful to improve CVO status reporting.

Stop maintaining RRI condition and remove it if previously present.

Previously we enabled this experimental behavior for the `UpgradeStatus` cluster-wide feature gate. We want to use the `UpgradeStatus` feature as a "vehicle" to promote the `oc adm upgrade status` CLI but we do not want to promote the RRI with it.

I do not remove the rest of the reconciliation_issues.go code that creates the content we were feeding into the condition because I have ideas about how to make that functionality useful to improve CVO status reporting.
@openshift-ci-robot openshift-ci-robot added the jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. label Jul 22, 2025
@openshift-ci-robot
Copy link
Contributor

openshift-ci-robot commented Jul 22, 2025

@petr-muller: This pull request references OTA-1578 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the task to target the "4.20.0" version, but no target version was set.

In response to this:

Stop maintaining RRI condition and remove it if previously present.

Previously we enabled this experimental behavior for the UpgradeStatus cluster-wide feature gate. We want to use the UpgradeStatus feature as a "vehicle" to promote the oc adm upgrade status CLI but we do not want to promote the RRI with it.

I do not remove the rest of the reconciliation_issues.go code that creates the content we were feeding into the condition because I have ideas about how to make that functionality useful to improve CVO status reporting.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@openshift-ci openshift-ci bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jul 22, 2025
@petr-muller
Copy link
Member Author

/test ?

Copy link
Contributor

openshift-ci bot commented Jul 22, 2025

@petr-muller: The following commands are available to trigger required jobs:

/test e2e-agnostic-operator
/test e2e-agnostic-operator-techpreview
/test e2e-agnostic-ovn
/test e2e-agnostic-ovn-upgrade-into-change
/test e2e-agnostic-ovn-upgrade-into-change-techpreview
/test e2e-agnostic-ovn-upgrade-out-of-change
/test e2e-agnostic-ovn-upgrade-out-of-change-techpreview
/test e2e-aws-ovn-techpreview
/test e2e-hypershift
/test e2e-hypershift-conformance
/test gofmt
/test images
/test lint
/test okd-scos-images
/test unit
/test verify-deps
/test verify-update
/test verify-yaml

The following commands are available to trigger optional jobs:

/test e2e-agnostic-operator-devpreview
/test e2e-extended-tests
/test okd-scos-e2e-aws-ovn

Use /test all to run the following jobs that were automatically triggered:

pull-ci-openshift-cluster-version-operator-main-e2e-agnostic-operator
pull-ci-openshift-cluster-version-operator-main-e2e-agnostic-operator-devpreview
pull-ci-openshift-cluster-version-operator-main-e2e-agnostic-ovn
pull-ci-openshift-cluster-version-operator-main-e2e-agnostic-ovn-upgrade-into-change
pull-ci-openshift-cluster-version-operator-main-e2e-agnostic-ovn-upgrade-out-of-change
pull-ci-openshift-cluster-version-operator-main-e2e-aws-ovn-techpreview
pull-ci-openshift-cluster-version-operator-main-e2e-hypershift
pull-ci-openshift-cluster-version-operator-main-e2e-hypershift-conformance
pull-ci-openshift-cluster-version-operator-main-gofmt
pull-ci-openshift-cluster-version-operator-main-images
pull-ci-openshift-cluster-version-operator-main-lint
pull-ci-openshift-cluster-version-operator-main-okd-scos-e2e-aws-ovn
pull-ci-openshift-cluster-version-operator-main-okd-scos-images
pull-ci-openshift-cluster-version-operator-main-unit
pull-ci-openshift-cluster-version-operator-main-verify-deps
pull-ci-openshift-cluster-version-operator-main-verify-update

In response to this:

/test ?

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@petr-muller
Copy link
Member Author

/test e2e-agnostic-ovn-upgrade-into-change-techpreview

Copy link
Member

@wking wking left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/lgtm

@openshift-ci openshift-ci bot added the lgtm Indicates that a PR is ready to be merged. label Jul 22, 2025
Copy link
Contributor

openshift-ci bot commented Jul 22, 2025

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: petr-muller, wking

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@petr-muller
Copy link
Member Author

/test e2e-agnostic-ovn-upgrade-into-change-techpreview
/retest-required

@petr-muller
Copy link
Member Author

petr-muller commented Jul 23, 2025

/test e2e-agnostic-operator-devpreview

Failed to install DevPreview but unlikely to be caused by this PR, I will rerun to obtain an additional data point.

@petr-muller
Copy link
Member Author

petr-muller commented Jul 23, 2025

OTA-1578

Inspecting TechPreview jobs running on OCP with the changes in this PR. The ReconciliationIssues condition is expected to be absent after installation and upgrade.

Installation TP job

Link: https://prow.ci.openshift.org/view/gs/test-platform-results/pr-logs/pull/openshift_cluster-version-operator/1216/pull-ci-openshift-cluster-version-operator-main-e2e-aws-ovn-techpreview/1947683862796570624

$ curl --silent https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/test-platform-results/pr-logs/pull/openshift_cluster-version-operator/1216/pull-ci-openshift-cluster-version-operator-main-e2e-aws-ovn-techpreview/1947683862796570624/artifacts/e2e-aws-ovn-techpreview/gather-extra/artifacts/clusterversion.json | \
                  jq '.items[0].status.conditions | length'
7

$ curl --silent https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/test-platform-results/pr-logs/pull/openshift_cluster-version-operator/1216/pull-ci-openshift-cluster-version-operator-main-e2e-aws-ovn-techpreview/1947683862796570624/artifacts/e2e-aws-ovn-techpreview/gather-extra/artifacts/clusterversion.json | \
                  jq '.items[0].status.conditions[] | .type'
"RetrievedUpdates"
"Upgradeable"
"ImplicitlyEnabledCapabilities"
"ReleaseAccepted"
"Available"
"Failing"
"Progressing"

Update TP job:

Link: https://prow.ci.openshift.org/view/gs/test-platform-results/pr-logs/pull/openshift_cluster-version-operator/1216/pull-ci-openshift-cluster-version-operator-main-e2e-agnostic-ovn-upgrade-into-change-techpreview/1947803163239124992

$ curl --silent https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/test-platform-results/pr-logs/pull/openshift_cluster-version-operator/1216/pull-ci-openshift-cluster-version-operator-main-e2e-agnostic-ovn-upgrade-into-change-techpreview/1947803163239124992/artifacts/e2e-agnostic-ovn-upgrade-into-change-techpreview/gather-extra/artifacts/clusterversion.json | \
                  jq '.items[0].status.conditions | length'
7

$ curl --silent https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/test-platform-results/pr-logs/pull/openshift_cluster-version-operator/1216/pull-ci-openshift-cluster-version-operator-main-e2e-agnostic-ovn-upgrade-into-change-techpreview/1947803163239124992/artifacts/e2e-agnostic-ovn-upgrade-into-change-techpreview/gather-extra/artifacts/clusterversion.json | \
                  jq '.items[0].status.conditions[] | .type'
"RetrievedUpdates"
"Upgradeable"
"ImplicitlyEnabledCapabilities"
"ReleaseAccepted"
"Available"
"Failing"
"Progressing"

@petr-muller
Copy link
Member Author

Baseline

Inspecting TechPreview jobs running on OCP without the changes in this PR (jobs from unrelated PRs). The ReconciliationIssues condition is expected to be present.

Installation TP job from #1215

Link: https://prow.ci.openshift.org/view/gs/test-platform-results/pr-logs/pull/openshift_cluster-version-operator/1215/pull-ci-openshift-cluster-version-operator-main-e2e-aws-ovn-techpreview/1947963318819885056

$ curl --silent https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/test-platform-results/pr-logs/pull/openshift_cluster-version-operator/1215/pull-ci-openshift-cluster-version-operator-main-e2e-aws-ovn-techpreview/1947963318819885056/artifacts/e2e-aws-ovn-techpreview/gather-extra/artifacts/clusterversion.json | \
                  jq '.items[0].status.conditions | length'
8

$ curl --silent https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/test-platform-results/pr-logs/pull/openshift_cluster-version-operator/1215/pull-ci-openshift-cluster-version-operator-main-e2e-aws-ovn-techpreview/1947963318819885056/artifacts/e2e-aws-ovn-techpreview/gather-extra/artifacts/clusterversion.json | \
                  jq '.items[0].status.conditions[] | select(.type=="ReconciliationIssues")'
{
  "lastTransitionTime": "2025-07-23T11:19:03Z",
  "message": "No issues found during reconciliation",
  "reason": "NoIssues",
  "status": "False",
  "type": "ReconciliationIssues"
}

Update TP job from #1206:

Link: https://prow.ci.openshift.org/view/gs/test-platform-results/pr-logs/pull/openshift_cluster-version-operator/1206/pull-ci-openshift-cluster-version-operator-main-e2e-agnostic-ovn-upgrade-into-change-techpreview/1947679666575773696

$ curl --silent https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/test-platform-results/pr-logs/pull/openshift_cluster-version-operator/1206/pull-ci-openshift-cluster-version-operator-main-e2e-agnostic-ovn-upgrade-into-change-techpreview/1947679666575773696/artifacts/e2e-agnostic-ovn-upgrade-into-change-techpreview/gather-extra/artifacts/clusterversion.json | \
                  jq '.items[0].status.conditions | length'
8

$ curl --silent https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/test-platform-results/pr-logs/pull/openshift_cluster-version-operator/1206/pull-ci-openshift-cluster-version-operator-main-e2e-agnostic-ovn-upgrade-into-change-techpreview/1947679666575773696/artifacts/e2e-agnostic-ovn-upgrade-into-change-techpreview/gather-extra/artifacts/clusterversion.json | \
                  jq '.items[0].status.conditions[] | select(.type=="ReconciliationIssues")'
{
  "lastTransitionTime": "2025-07-22T18:08:34Z",
  "message": "No issues found during reconciliation",
  "reason": "NoIssues",
  "status": "False",
  "type": "ReconciliationIssues"
}

@petr-muller
Copy link
Member Author

Inspecting the baseline jobs shows the CV has 8 conditions including ReconciliationIssues.
Inspecting the jobs on this PR shows the CV has 7 conditions and does not have ReconciliationIssues.

Looks good.

@petr-muller
Copy link
Member Author

/test e2e-agnostic-ovn-upgrade-out-of-change-techpreview

Copy link
Contributor

openshift-ci bot commented Jul 23, 2025

@petr-muller: all tests passed!

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

petr-muller added a commit to petr-muller/api that referenced this pull request Jul 23, 2025
This is a request for an [exception](https://github.com/openshift/enhancements/blob/master/dev-guide/featuresets.md#what-if-i-dont-have-automated-testing-reporting-into-component-readiness) in the feature promotion process.

Proposed promotion process:

1. Promote `UpgradeStatus` gate here. This does not actually put any functional change into customers' hands.
2. Promotion will allow tests tagged with `UpgradeStatus` to run in Default update jobs
3. Update-time tests will execute and feed results into Sippy feature view
4. The feature view will need to be monitored manually to meet the criteria otherwise enforced by `verify-feature-promotion`
5. Once that happens, _actually_ promote the feature through openshift/oc#2063

Why is exception needed:

The feature promoted under `UpgradeStatus` gate identifier is actually entirely client-side implementation of the `oc adm upgrade status` command. Promoting the feature in o/api has no effect on any functionality in the cluster; nothing reads and depends on this identifier in the cluster (as of Jul 23, with the exception of the about to be removed behavior, tracked in OTA-1578 / openshift/cluster-version-operator#1216 expected to merge soon). The actual "promotion" of the feature would be done in the o/oc repository with a PR such as openshift/oc#2063 (intentionally `/hold` for now because of its "feature promotion" semantics).

We wanted to utilize the `UpgradeStatus` gate to pursue the data-driven promotion process, but the process does not seem to accommodate features tied to OCP updates. Features are expected to accumulate test coverage in TechPreview (TP) jobs, but there are no TP jobs that exercise a cluster update. Minor version update TP cannot exist by definition (`TechPreviewNoUpgrade`). Patch version update jobs can exist in theory, but is likely that OCP cannot realistically maintain such jobs because teams are allowed to make update-breaking changes in TP (such as promotion where a v1alpha1 API will be replaced with a v1 one) until the version is GA. That means update-related features cannot easily accumulate sufficient test data in available TechPreview jobs (we may need to address this gap in the future because the exception process proposed here will not succeed for cluster-side behavior).

For this feature specifically, running in TechPreview jobs would actually increase risk of problems after promotion. The conditional behavior is entirely client-side; testing it against TP clusters carries a risk of inadvertedly depending on some cluster-side TP-only feature. Such feature could be missing in Default clusters and the client-side behavior can break when run against a Default cluster. For CLI-side features, it seems more appropriate to test TP behavior against a Default cluster instead. This is possible, but non-TP jobs will not execute tests tagged with a TP feature gate name such as `FeatureGate`.

How are regressions prevented:

- Promoting `UpgradeStatus` in o/api has no risks because it does not change any behavior except test selection
- Running candidate CLI behavior against Default clusters actually provides better quality signal than against TP clusters
- Tests tagged with `UpgradeStatus` will need better scrutiny before merge because they will run in Default jobs and not just TP jobs
- The actual promotion in o/oc is still gated by data evidence of passing tests, in spirit with the usual promotion process
@dis016
Copy link

dis016 commented Jul 24, 2025

/label qe-approved

@openshift-ci openshift-ci bot added the qe-approved Signifies that QE has signed off on this PR label Jul 24, 2025
@openshift-ci-robot
Copy link
Contributor

openshift-ci-robot commented Jul 24, 2025

@petr-muller: This pull request references OTA-1578 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the task to target the "4.20.0" version, but no target version was set.

In response to this:

Stop maintaining RRI condition and remove it if previously present.

Previously we enabled this experimental behavior for the UpgradeStatus cluster-wide feature gate. We want to use the UpgradeStatus feature as a "vehicle" to promote the oc adm upgrade status CLI but we do not want to promote the RRI with it.

I do not remove the rest of the reconciliation_issues.go code that creates the content we were feeding into the condition because I have ideas about how to make that functionality useful to improve CVO status reporting.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@openshift-merge-bot openshift-merge-bot bot merged commit 827d841 into openshift:main Jul 24, 2025
19 checks passed
@openshift-bot
Copy link
Contributor

[ART PR BUILD NOTIFIER]

Distgit: cluster-version-operator
This PR has been included in build cluster-version-operator-container-v4.20.0-202507240745.p0.g827d841.assembly.stream.el9.
All builds following this will include this PR.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. lgtm Indicates that a PR is ready to be merged. qe-approved Signifies that QE has signed off on this PR
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants